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Abstract 

Wc present results of a search for Lyman break galaxies (LBGs) at z ~ 5 in a 618 squarc-arcmin field 
including the Hubble Deep Field-North (HDF-N) taken by Subaru Prime Focus Camera. Utilizing the 
published redshift data of the HDF-N and its flanking fields, the color selection criteria are chosen so that 
LBGs are picked out most efficiently and least contaminated by foreground objects. The numbers of LBG 
candidates detected are 96 in 23.0 < I c mag < 24.5, and 310 in 23.0 < I c < 25.5. There is a hint of the 
deficiency of bright blue galaxies, although it is not as clear as has been suggested for LBGs at z ~ 3 to 
4. With 305 LBG candidates in a 575 square-arcmin field, the rest-frame UV luminosity function of LBGs 
at 4.4 < z < 5.3 is derived statistically, by considering both contamination by objects at the intermediate 
redshift and incompleteness of the survey. The fraction of contamination is estimated to be ~ 50% in the 
faintest magnitude range. The completeness of the survey is ^80% at the bright part of the sample, and 
~ 20% in the faintest magnitude range (25.0 < I c < 25.5). The luminosity function of LBG candidates at 
z ~ 5 does not show a significant difference from those of LBGs at z ~ 3 and 4, though there might be 
a slight decrease in the fainter part. The UV luminosity density within the observational limit is 0.56 - 
0.69 times smaller than that obtained for LBGs at z *~ 3, depending on the adopted cosmology and the 
integration range of the luminosity function. This decrease in the UV luminosity at z ~ 5 compared to that 
at z ~ 3 is due to the smaller number density of faint galaxies at z ~ 5. The similarity of the luminosity 
functions at redshifts 5 to 3 implies that most of the LBGs at z ~ 5 should have faded out at z ~ 3 and the 
LBGs at z ~ 5 are different galaxies from those seen at z ~ 3, if we take face values for ages of the LBGs at 
z ~ 3 obtained by the SED fitting in which a continuous star formation in an individual galaxy is assumed. 
However, if the star formation in LBGs is sporadic, the similarity of the luminosity function at z ~ 3 and 
5 would be explained. Such sporadic star formation has been suggested by hydrodynamical simulations 
and semi-analytic models with collisional starbursts, and the trend of the cosmic star formation history 
predicted by these studies resembles to that estimated from an integration of the UV luminosity density 
within the observational limit. 
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1. Introduction 

In the past several years, observational studies of galaxy formation and evolution have progressed remarkably. One 
of the most important progresses is that statistical nature of high redshift galaxies has been revealed through a finding 
of many normal (non-AGN) galaxies at high redshift, which has been enabled by the broadband color selection method 
pioneered by Steidel and his collaborators (e.g., Steidel and Hamilton 1992; Steidel et al. 1996a). Since the method 
employs only broad band imaging, it does not need a huge amount of telescope time such as optical spectroscopy of 
faint galaxies, and it efficiently finds galaxies at a targeted redshift. In fact, the redshifts of more than 80% of the 
galaxies selected by this method (called Lyman break galaxies) are confirmed by optical spectroscopy (Steidel et al. 
1996a; Steidel et al. 1999). 



Based on data collected at Subaru Telescope, which is operated by the National Astronomical Observatory of Japan. 
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Since their first discovery in the late 1990s, the Lyman break galaxies (LBGs) at z ~ 3 have been extensively 
investigated in various ways and knowledge on their nature has been accumulated. Their rest UV spectra resemble 
those of nearby active star-forming galaxies (Steidel et al. 1996b). On-going star formation rates estimated from UV 
luminosities and from emission lines in rest-frame optical band are up to several tens of M Q yr _1 (e.g., Steidel et al. 
1996b; Pettini et al. 2001). Their stellar population has been investigated by comparing the optical to near-infrared 
spectral energy distributions (SEDs) with stellar population synthesis models (Sawicki and Yee 1998; Papovich et 
al. 2001: Shaplcy ct al. 2001). It is found that most of the LBGs at z ~ 3 are well fitted by models dominated by young 
stellar population with ages ranging from 10 Myr to 1 Gyr with moderate dust extinction (E(B — V) ~ 0.1 - 0.4) and 
their typical stellar mass is estimated to be ~ 10 10 M Q , similar to a typical bulge mass of a present-day giant galaxy. 
Metal abundance in ionized gas is estimated through rest optical emission line ratios to be sub-solar similar to those 
of Large and Small Magellanic Clouds (Tcplitz ct al. 2000; Pettini et al. 2001). The clear signal of spatial clustering of 
the LBGs at z ~ 3 has been reported (Steidel et al. 1998; Giavalisco et al. 1998), and the comparison with the models 
of cold dark matter (CDM) halos suggests that their clustering properties of the LBGs can be ascribed to the biased 
galaxy formation (Giavalisco and Dickinson 2001). 

Steidel et al. (1999) extend their ground-based LBG search to z ~ 4. They surveyed a 828 arcmin 2 area in total 
which is comprised of 10 fields and found 244 galaxies with Tab < 25.0 mag. The luminosity function of the LBGs at 
z ~ 4 remarkably resembles that at z ~ 3, indicating that no significant evolution of luminosity function occurs from 
z = 4 to z = 3 (about 0.6 Gyr for the cosmology with £1m = 0.3, Oa = 0.7 and Hq = 65 km s _1 Mpc -1 ). Ouchi ct 
al. (2001) also report the result of their own LBG survey for z ~ 4 in the field of SUBARU/XMM Deep Survey. They 
found that the luminosity function almost agrees with that obtained by Steidel et al. (1999) and that the significant 
clustering is also seen. 

The LBGs make the statistical sample of UV luminous objects at high redshift. Since the UV luminosity density 
is considered to be correlated with star formation rate per unit volume, the LBGs have opened up a possibility to 
examine the star formation history in the early universe (Madau et al. 1996, 1998). Steidel et al. (1999) found that 
the UV luminosity density at z ~ 4 is almost comparable to that at z ~ 3. If the correction for the dust extinction is 
applied, the star formation densities at z — 3 and 4 are comparable to the value at z = 1 — 2. The similarity of the 
luminosity function of LBGs at z = 3 and at z = 4 and the rather flat (extinction corrected) star formation density 
history in 1 < z < 4 arc quite interesting and may be a key property to understand star formation history of galaxies. 
As mentioned above, the ages of the LBGs at z = 3 are estimated to be 10 Myr to 1 Gyr with a median value of 
0.1-0.3 Gyr based on the SED fitting analyses by assuming a continuous star formation. If these ages can be regarded 
as time intervals from the onset of the first star formation, the LBGs at z ~ 3 should be very young and most of them 
should form at z < 4 (e.g., Papovich et al. 2001). Accordingly, the star formation density contributed by the LBGs at 
z ~ 3 should be lower by a factor of about 10 at z > 4 (Ferguson ct al. 2002) and the LBGs at z ~ 3 must be different 
from the LBGs at z > 4. 

By using semi-analytic models based on the hierarchical clustering scenario, several authors have tested whether 
the observed number densities and star formation densities at z ~ 3 and z ~ 4 can be reproduced (e.g., Baugh et 
al. 1998; Somerville et al. 2001; Balland et al. 2002). Somerville et al. (2001) examine the nature of these LBGs using 
semi-analytic models including starburst induced by collision of galaxies as well as continuous star formation. They 
claim that their model with collisional starburst reproduces the observed luminosity functions of LBGs both at z = 3 
and z — 4 with a reasonable amount of dust extinction, and the estimated star formation density is claimed to be 
compatible with observations. On the other hand, the model with constant star formation efficiency (which does not 
include collisional starburst) fails to reproduce the observed luminosity function of LBGs at z ~ 3, and star formation 
density declines rapidly at z > 2. 

Pushing these studies back to the earlier universe and disclosing the nature of higher redshift galaxies is an obvious 
next target. The extension of the observational data toward higher redshift provides a stronger constraint on physical 
processes of the galaxy evolution through the comparison with theoretical models. It is also indispensable to study 
the epoch of the generation of the first galaxies. The study at the higher rcdshifts has a further importance because it 
is approaching to a suggested epoch of the cosmological rcionization (z > 6; Becker et al. 2001). The number densities 
of the LBGs may provide a constraint on determining the contribution of the star forming galaxies to the ionizing 
background radiation (Madau et al. 1999). The discoveries of several star forming galaxies at z > 5 have been already 
reported (e.g., Dey et al. 1998; Weymann et al. 1998; Spinrad et al. 1998; Dawson et al. 2001; Ellis et al. 2001; Dawson 
ct al. 2002). Their properties are important in the context of the investigation of the nature of galaxies at high redshift. 
However, since there has not been a volume-limited large sample, statistical properties of galaxies at z ~ 5 remain 
unknown. Finding many galaxies at z ~ 5 systematically and studying their statistical nature is necessary for the 
first step. We should carry out a survey with a sufficiently large search area at a deep limiting magnitude. However, 
it is not easy in technical aspects. The distance to the objects are getting larger and thus apparent magnitude and 
surface brightness are getting fainter. Consequently a number density at an apparent magnitude decreases rapidly. In 
addition, the observing band shifts to the redder wavelength region where OH sky emission increases the background 
noise. Therefore a large aperture of a telescope and a wide field of view are required to perform this task. 
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Here we present the first results of our survey of LBGs at z ~ 5 using the Subaru Prime Focus Camera (Suprime- 
Cam; Miyazaki et al. 2002), utilizing its unique capability of a wide field of view among 8m class telescopes. We 
choose the sky area including the Hubble Deep Field-North (HDF-N; Williams et al. 1996), because there is plenty 
of information of galaxies with a wide redshift range (0 < z < 6). This database is quite powerful to examine our 
method for searching for galaxies at z ~ 5 and to obtain a reliable sample of candidates, under the circumstances that 
considerably time consuming optical follow-up spectroscopy must be carried out after making a candidate catalog. 
The field is also the sky area where many deep surveys in variety of wavelengths such as X-ray, infrared and radio. 
We will be able to utilize these data to investigate nature of LBG population in other wavebands. 

This paper is organized as follows. In section 2, the observations and procedures of data reduction are described. In 
section 3, criteria for finding LBGs at z ~ 5 and the number of objects detected are presented. Properties of the LBG 
candidates are also described. Then in section 4, we quantify the corrections needed to apply to the observed data 
for deriving the luminosity function statistically, and the resultant luminosity function of LBGs at z ~ 5 is presented. 
In section 5, implications of our findings to the evolution process of LBGs and the evolution of the UV luminosity 
density over the cosmic time scale is discussed. The contribution of LBGs at z ^ 5 to the UV ionizing field is also 
mentioned. In section 6 we give a summary and conclusion of our findings. 

We use a set of cosmological parameters of Ho = 65 km s _1 Mpc -1 , Qm = 0.3, and fl\ = 0.7 as a primary set, and 
in some cases we also use an Einstein-de Sitter universe model with Ho — 50 km s _1 Mpc -1 , Om = 1.0 and fl\ = 
which had been commonly used as a standard set of cosmological parameters. The magnitude system is based on 
Vega magnitude except for the cases otherwise noted. 

2. Observation and Data Reduction 

2.1. Filter Selection 

Prior to the observation, the combination of the filters suited to isolate the z ~ 5 population was explored using 
mock galaxy spectra constructed with stellar population synthesis codes. Although it is not obvious that SEDs of 
LBGs at z ~ 5 arc similar to those at z ~ 3, we adopted model SEDs of LBGs at z ~ 5 to enclose various rest-frame UV 
SEDs of LBGs at z ~ 3. The model spectrum for young, actively star-forming galaxies, by which we intend to model 
the SED of LBGs with the bluest UV color, was constructed using the stellar synthesis code developed by Kodama 
and Arimoto (1997). We included the attenuation due to the inter-galactic medium following the prescription by 
Madau (1995). In order to have the model spectra delineate the redder SED of LBGs, we adopted to take into account 
the reddening effect by dust within the object itself using the extinction curve given by Calzetti et al. (2000) with 
E(B - V) from 0.0 to 0.4. In figure 1 we show the SEDs of the model spectra with E(B - V) = 0.0, 0.2 and 0.4, as 
well as the rest- frame photometric data points of LBGs at z ~ 3 in the HDF-N (Papovich et al. 2001). The model 
spectra reasonably cover the UV colors of the LBGs at z ~ 3. The adoption of the dust extinction is rather arbitrary 
to reproduce the redder colors of LBGs. The interpretation is, however, often used to derive color excesses in LBGs 
(e.g., Steidel et al. 1999). 

We also prepared model spectra of spiral and elliptical galaxies for examining the efficiency of eliminating galaxies 
at intermediate redshifts. As spectra of spiral galaxies we used those of Sbc and Scd galaxies derived by Coleman et 
al. (1980). As a spectrum of an elliptical galaxy we employed the model spectrum by Kodama and Arimoto (1997) 
which reproduces the color-magnitude relation of elliptical galaxies observed in the local universe. Then wc calculated 
observed spectra for each type of galaxy from z = to z = 6 with a step of 0.1. We did not consider any effect of 
evolution for all types of galaxies. 

A desirable combination of the filters should give the most distinct separation of the z ~ 5 population apart from the 
objects with smaller redshifts. We chose a combination of V, I c , and z'-bands. In the figure 2a, we show the two-color 
V — I c versus I c — z' diagram. It is seen that galaxies at z > 4.5 can be well separated from foreground objects in the 
diagram. The most critical contamination is caused by elliptical galaxies at z ~ 0.5 due to the 4000 A break and early 
type spirals at intermediate redshifts. 

Selecting i?-band instead of U-band may also be reasonable, because of the high quantum efficiency at the band. 
However, contamination by galaxies with 4000 A break at intermediate redshifts is more serious than the adopted 
filter combination. Another reason we did not choose i?-band is that the continuum depletion from the Lyman alpha 
forest should be heavier in U-band, and the clear Lyman break makes it easier to isolate LBGs in a two-color diagram. 

2.2. Observations 

The observations were carried out on 22nd and 23rd February, 2001 with Suprime-Cam attached to Subaru telescope 
at Mauna Kea. The Suprime-Cam consists of 10 mosaiced CCDs, each of which has a dimension of 2046 by 4090 
pixels with a pixel scale of 0."2. Each CCD covers a field of ~ 7' by ~ 13' and a typical gap between the CCDs is 
10" — 15". The details of the observation are summarized in table 1. Note that our observations were made before the 
upgrade of the CCD array configuration held on April 2001, and the one CCD chip (at the north-west corner) was not 
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Fig. 1. Model UV spectra of LBGs at z ~ 5, which are used for the filter selection and for the evaluation of the incompleteness of 
our survey via Monte Carlo simulation (in section 5.2). The bluest (extinction-free) spectrum is shown in the blue solid line. The 
green and red lines are spectra of the redder LBGs attenuated by dust with E(B — V) = 0.2 and 0.4, respectively. The attenuation 
by the intergalactic matter is taken into account (Madau et al. 1996). Points are photometric data of the LBGs in the HDF-N at 
z ~ 3 (Papovich et al. 2001); we intend to cover the various SEDs at z ~ 3 by model spectra. All flux densities are normalized at 
2000 A. In the upper side of the figure we show the effective band-passes of the Suprime-Cam with V, I c and z' filters, which are 
blucshifted as they work for an object at z = 5. 



Table 1. Details of the observation. 



Dates 

CCD dimension 
Pixel scale 

Field of view / Effective surveyed area[*] 
Filters 

Effective integration time 
Seeing 

Magnitude of 5tr detection limit [f] 



22 and 23 Feb. 2001 
9 CCDs with 2046 x 4090 pixels 
0.2'Vpixel 
836 arcmin 2 / 575 arcmin 2 



V 



I, 



21,600 sec. 
0".8 - 1".0 
28.5 mag. 



2,520 sec. 
0".7 - 1".0 
25.5 mag. 



3,100 sec. 
0".6 - 1".0 
25.2 mag. 



The effective surveyed area represents the area used to derive the luminosity function, 
t The magnitude of the faintest object that can achieve S/N of 5 with the typical aperture 
size corresponding to the area of the detected sources at the magnitude range (0."9 for V 
and z'-bands, l"2 for 7 c -band). 



working and another chip (at the south-west corner) had many bad columns. Thus 8 CCDs were effectively working, 
and about ~ 27' by ~ 27' field is covered by a exposure. 

The imaging observation was made pointed to the Hubble Deep Field-North (RA(2000) = 12 h 36 m 49?4, Dec(2000) 
= +62°12'58"). Images were taken with a small dithering (typically 20") with a typical exposure time of 1440 sec, 180 
- 240 sec, and 120 sec, for V, I c , and z'-bands, respectively for each exposure. A seeing size during the observations 
was typically ~ 0"9 and we discarded some frames with a median FWHM of stars larger than 1"0 for keeping the 
angular resolutions of the final images. Effective integration times were 21.6 ksec, 2.5ksec, and 3.1 ksec for V, I Cl and 
z'-bands, respectively. 

2.3. Data Reduction and Calibration 

The data reduction was carried out in a standard manner using IRAF 1 for basic data processing; bias was subtracted 
using counts in an overscan region, flat field frames were made by using normalized average of the object frames. The 
sensitivity differences between the CCD chips have been corrected using the relative count rates of the flat-fielded 
dome flat frames for all three bands. The background was subtracted by two dimensional polynomial fitting. Positions 
of the frames arc aligned by measuring the positions of the field stars appeared commonly in all frames and fitting 
fourth-order polynomial functions for coordinate conversion. Counts of each frame were adjusted by comparing the 
count-rate of the field stars common to all frames for every filter; we measured the counts of field stars in each frame, 
then the images were divided by the median of the count ratios of stars relative to a fiducial frame so that the count 



IRAF is distributed by the National Optical Astronomy Observatories, which are operated by the Association of Universities for 
Research in Astronomy, Inc., under cooperative agreement with the National Science Foundation. 
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Fig. 2. I c — z' versus V — I c diagrams, (a): Tracks of colors for model galaxies at redshift ranging from to 5.5. A blue (an orange) 
line represents a galaxy with the bluest (reddest) model spectrum, i.e., E(B — V) = 0.0 (E(B — V) = 0.4) (see figure 1). A green and 
cyan dashed lines refer to local spiral galaxies (green is for Sbc, and cyan is for Scd; Coleman et al. 1980). A red dashed line is for a 
local elliptical galaxy. No evolution is considered. The symbols are plotted at a redshift interval of 0.5, and the symbols at z > 4.5 
are enlarged. Star-symbols indicate the colors of AO - M9 stars calculated based on the library by Pickles (1998). Hatched region 
indicates the color criteria we adopted for z ~ 5 galaxies, (b): Colors of galaxies cataloged in the spectroscopic survey of galaxies 
in the HDF-N and its flanking fields by Cohen et al. (2000) and identified in our image. All galaxies with the redshift range z < 4.5 
are plotted. Tracks for model spectra are also shown, (c): Same as (b), but for galaxies in the HDF-N for which the photometric 
redshifts are given by Fernandez-Soto et al. (1999) as z < 4.5. (d) — (f): The color distribution of the sources extracted from our 
image. Panels (d), (c) and (f) show objects with I c < 23.5, 23.5 < I c < 24.5 and 24.5 < I c < 25.5, respectively. Circles are objects 
detected in all of V, I c , and z'-bands, and arrows are those detected in I c and z 1 but not in V-band. Lower limits of V — I c colors 
are given for them. 
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of stars are almost constant among frames. 2 The images were convolved so that the median FWHM of stellar sources 
match within 10% for all frames in three bands. 

Photometric calibrations for V and 7 c -band images were made using images of photometric standard stars in 
Landolt (1992). The accuracy was ~ 0.02 mag for V and ~ 0.08 mag for I c . For the photometric calibration of the 
z-band images, we employed two ways both based on the I c — z' color. We first set an arbitrary zero point for z' 
magnitude. In the first method, we examined the distribution of V — I c and I c — z' colors of field stars exposed on our 
images. Next we calculated the I c — z' colors to be observed with the system sensitivity of Suprime-Cam for the stars 
in the stellar flux library by Pickles (1998). We can determine the magnitude zero point for z'-band by comparing the 
distribution of these colors. Another method is to use the spectral energy distribution of the galaxies in the HDF-N 
and its flanking fields. We used the galaxies for which the published V, I and J-band magnitudes are available and 
estimated their I c — z' colors by linear interpolation. Comparison with the colors in our image gave the magnitude zero 
point. We adopt the magnitude zero point determined by the latter method. The z 1 magnitude zero points derived by 
these two methods agrees within ~ 0.1 mag in the color range of I c — z' between ~ and 1. The z'-band magnitude is 
also based on Vega. We used the flux density table of Vega given by Castelli and Kurucz (1994), and applied z'(AB) 
- z'(Vega) = 0.5505. 

2.4- Source Detection 

Prior to the extraction of the astronomical objects from the final imaging data, we set masks for source detection 
around the bright objects in order to avoid the misidentification of the noise or tails of the bright objects as an 
individual object. We also removed the edge regions of each CCD chip from the object detection because of their low 
signal-to-noise ratio caused by dithering imaging. In total 26.1 % of the CCD chip area is masked, and the effective 
area observed is 618.6 arcmin 2 (575.0 arcmin 2 without the CCD chip with many bad columns). 

We used the software SExtractor (Bertin and Arnouts 1996) for the source detection and photometry. The primary 
parameters for the object detection are the threshold of count above which the pixel is treated as a part of an object 
and the minimum number of connected pixels to be classified as an object. Wc extensively examined these parameters 
by comparing the detected sources with the objects in the HDF catalog version 2 (Williams et al. 1996) brighter than 
Jab (814) — 27 , to minimize both the rate of noise contamination and the rate of missing objects. We also tested some 
spatial smoothing filters and adopted a Gaussian smoothing filter with a FWHM of 3 pixels for I c and z'-bands. 

The best set of parameters with which the detection rate of faint objects and the noise contamination are compro- 
mised is that the thresholds are 1.3, 1.0, 1.0 times of pixel-based background noise for V, I c and z', respectively and 
to have at least 4 connected pixels above threshold in each band. We regard sources in different bands as the same 
object whose pixel coordinates are within 1.0". We pick out objects only if they are detected both in I c and z'-bands. 
The number of sources thus selected is 29,094 for I c < 25.0 and 35,362 for I c < 25.5 in the 9 CCD chips. Although 
the adopted detection criteria seem to be loose, i.e., seem to tend to pick out many noise features, the criterion of two 
band detection removes spurious sources efficiently. It would be clarified by the comparison of the detected objects to 
the HST/WFPC2 data of HDF-N (Williams et al. 1996), which have deeper 1/(606) and 7(814) images. There is no 
misidentification of noise as an object in I c < 24.5 and the number of the noise misidentification in 24.5 < I c < 25.0 
is two among 55 extracted sources within the HDF-N region. Even in the magnitude range 25 < I c < 25.5, which is 
the faintest range we use, the rate of misidentification of noise is not so large; it is ~ 17%. Many of these spurious 
sources locate close to the objects which are relatively bright (but fainter than masked objects). These objects degrade 
quality of flat field around them and presumably cause misidentification. The number of selected sources are robust for 
changes of the detection parameters; if we adopt 10 pixels for the minimum connection of pixels for V and 7 c -bands, 
we only lose ~ 2% of the objects at the faintest magnitude range. The vast majority of selected objects have area 
larger than 10 pixels in 7 c -bands. The fraction of selected objects in our image among the HDF-N catalog is getting 
smaller as the magnitude goes fainter; ~ 100 % for Jab (814) < 25, ~ 67 % for 25 < Jab (814) < 25.5, ~ 28 % for 
25.5 < Jab (814) < 26.0. The detection rates in other CCD chips do not change much as described in section 4.1. The 
number counts in 7 c -band give 4.0 x 10 4 deg" 2 mag -1 for 23.0 < I c < 23.5, 5.6 x 10 4 for 23.5 <I C < 24.0, 7.4 x 10 4 for 
24.0 < I c < 24.5, 8.3 x 10 4 for 24.5 < I c < 25.0 and 7.7 x 10 4 for 25.0 < I c < 25.5. If we correct the incompleteness of our 
source detection, the number counts of our data agree well with those appeared on previous studies (e.g., Tyson 1988; 
Williams et al. 1996). 

Photometry of the detected objects was also made using SExtractor. We use the value "best magnitude" in 
SExtractor for I c magnitude and a l."6 diameter aperture magnitude for V — I c and I c — z' colors. The photometric 
accuracies were estimated by putting many artificial objects into our final images, executing detection and photometry 

2 In this procedure, we found that although the variance of the flux ratios is stable and is well correlated with the airmass of the observed 
sky region on 23rd Feb, the flux ratio varies largely and the correlation of the flux ratio and the airmass is weaker on 22nd Feb. This 
suggests that the sky condition of the second observing night was better than the first night. Thus we selected one frame per filter 
as a fiducial frame, which was taken on the second night and the observed time is close to that of the exposures for standard stars. 
We adjusted the count-rate of the frames taken on the first night and other frames taken on the second night according to the fiducial 
frame. 
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with the same manner for the observed data and compare returned value with original magnitude. The sizes and 
apparent magnitudes of the artificial objects were adjusted so that the obtained magnitude and size distribution is 
similar to that of real data. The rms errors of I c -band "best magnitude" for objects with 24.0 <I C < 24.5, 24.5 <I C < 25.0 
and 25.0 < I c < 25.5 were 0.28, 0.35 and 0.41 mag, respectively. These error values are roughly consistent with the 
errors of I c magnitude estimated from the comparison with the magnitude for the objects in the HDF-N region. 
Photometric errors for V^-band data are 0.22, 0.31 and 0.33 mag for 26 < V < 26.5, 26.5 < V < 27.0 and 27.0 < V < 27.5, 
respectively, and for z'-band data, 0.39 mag, 0.45 mag and 0.49 mag for 23.5 < z' < 24.0, 24.0 < z' < 24.5 and 
24.5 < z' < 25.5, respectively. The errors in V — I c and I c — z' colors, which were measured with a 1"6 aperture 
magnitude, are smaller than the errors in the total magnitude. The rms errors of background noise in 7 c -band, which 
were measured at randomly selected positions in our image with a 1"6 aperture, were 0.08 mag and 0.28 mag for 
I c = 24.0 and 25.5, respectively. For the l/-band data, they were 0.07 mag and 0.25 mag for V = 26.0, and 27.5 
respectively, and for the z'-band data, they were 0.18 mag and 0.40 mag for z' = 24.0 and 25.0, respectively. 

The astrometry for detected objects were made using stars in our field-of-view listed in APM sky catalog (Irwin 
ct al. 1994). We added stars from USNO-A2.0 (Monet et al. 1998) to increase the number of stars for calculation 
of conversion coefficients. There is no systematic error larger than 0"3 between these two catalogues in our region. 
We measured the positions of the stars in our / c -band final image and fitted a fourth-order polynomial function to 
minimize the x 2 of the positional errors, for conversion of positions in our images to equatorial coordinates. The 
relative positions of CCD chips were also adjusted to provide a minimum error for the coordinate conversion. The 
achieved internal positional accuracy was *~ 0"4 rms for most part of the image, and ~ 1" at one CCD chip with many 
bad columns at the south-west corner. 

3. Lyman Break Galaxy Candidates at z ~ 5 

3.1. Selection Criteria 

One of the most serious problems in the study of statistical properties of Lyman break galaxies is a certain amount 
of contamination by objects located between us and objects at the targeted redshift (so called 'interlopers'). The 
uncertainty of the fraction of the contamination affects the reliability of statistical properties of the population, such 
as the luminosity function or the amplitude of the clustering. In figure 2a, we can see that a galaxy with an old stellar 
population (red dashed line) at z ~ 0.5 is close to the selection window for z ~ 5 galaxies; the 4000 A break in a galaxy 
at z ~ 0.5 mimics the color by the Lyman break at z ~ 5. 

In order to determine the selection criteria for galaxies at z ~ 5 as reliable as possible, we have utilized the previously 
published redshift data of the HDF-N and its flanking fields. We used the list of galaxies in the HDF-N and the flanking 
fields by Caltech Faint Galaxy Redshift Survey (CFGRS; Cohen ct al. 2000) and the catalog of photometric redshift 
of galaxies in the HDF-N by Fernandez-Soto et al. (1999). The magnitude limit is R ~ 25.5 mag for the CFGRS 
catalog and Jab (814) = 28.0 mag for the photometric redshift catalog. In our data we detect 566 galaxies out of 
the 671 galaxies in the CFGRS catalog and 317 galaxies out of the 477 galaxies with I c < 26 and at z < 4.5 in the 
photometric redshift catalogue. Most of the galaxies in the CFGRS catalog which have no counterpart in our image 
are located outside of our final image (between CCD chips or in the masked regions). Galaxies listed in the catalog 
by Fernandez-Soto et al. (1999) but not in our image are objects fainter than our limiting magnitude. In figure 2b 
and 2c, we plot the colors (J c — z' and V — I c ) of thus identified galaxies at z < 4.5 in the CFGRS sample and in the 
photometric redshift sample, respectively. 

Considering the distribution of the foreground galaxies in the two color diagrams we adopt the selection criteria for 
galaxies at z ~ 5 as 



V-I c >2.0, (1) 
and 

V-I c >7{I c -z')-0.1. (2) 

The selection window defined by these criteria is shown as a hatched region in figure 2. There is no galaxy in the 
selection window in the spectroscopic redshift database with z < 4.5 and only two galaxies in the photometric redshift 
database estimated to be at z < 4.5. These two galaxies are HDF 3—0238.0 and HDF 4—0505.1. The I c magnitude 
of HDF 3—0238.0 in our image is 24.8 mag and its position in the two color diagram is close to the boundary of 
our selection criteria. It is classified as an elliptical galaxy at z = 0.92 by Fernandez-Soto et al. (1999) and has a 
round shape both in drizzled HDF-N image and in our 7 c -band image. The estimated redshift of HDF 4—0505.1 
by Fernandez-Soto et al. (1999) is 1.44. It is associated with a more extended object HDF 4—0505.0, for which the 
photometric redshift is not available and it has similar colors to HDF 4—0505.1. Our 7 c -band image cannot resolve 
these two objects. These objects could be interlopers which happen to enter our color selection window. 
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In figure 2d-f, the color distribution of the all detected objects is shown. For objects which are not detected in 
lAband, we assign the lower limit of V — I c color using the limiting V magnitude of 28.5 mag. We regard such objects 
(detected in I c and z'-bands, but not in y-band) as candidates at z ~ 5, if they enter the selection window. There are 
objects with V — I c and I c — z' colors whose colors do not match any type of model galaxies or stars. For example, 
about 4% of all objects have I — z' < —0.1. The fraction of such objects with blue I — z' color increases in fainter 
magnitude. Although some of their colors may be affected by emission lines, we suppose most of them have redder 
intrinsic colors but are placed due to photometric errors. 

Among the 35,362 detected sources, 321 objects with I c < 25.5 fall within this selection window. There are 11 objects 
with 20.0 < I c < 23.0. Most of them lie very close to the boundary of our color selection criteria, and seven objects 
have neighbours within 5". All of them are clearly visible in our l^-band image and have round shapes, suggesting 
that they are elliptical galaxies with old stellar population at intermediate redshift. They are also unusually luminous 
if they are star- forming galaxies located at z ~ 5 (absolute UV magnitude would be ^ —22). We regard them as the 
interlopers which happen to fall within our color selection criteria due to photometric error and excluded them from 
our candidate list. In figure 3, we show V, I c and z'-band images of the representative objects in our LBG candidates 
in an order of I c magnitude. 

3.2. Cross-identification with Galaxies in the HDF-N and Its Flanking Fields 

We find six LBG candidates in the HDF-N region. The range of the I c magnitude of these objects are from 24.6 
to 25.4. One object HDF 4—625.0 is spectroscopically confirmed to be at a redshift of 4.58, and for three objects 
photometric rcdshifts arc provided by Fernandcz-Soto et al. (1999). The appearances of these four objects with redshift 
information in the HDF-N drizzled images, as well as in the Suprime-Cam images, are shown in figure 4. Among the 
three objects for which photometric redshifts are given, two galaxies (HDF 3—238.0 and HDF 4—505.0) have estimated 
redshifts of 0.92 and 1.44 respectively. These objects are detected in i?4 5 o-band, so these would be interlopers. The 
other object HDF 3—153.0, which appears to be a single object in our ground-based image, is identified as two resolved 
objects by Fernandez-Soto et al. (1999); one object has a redshift of 1.24 and the other 5.32. Since the latter object is 
0.9 mag brighter than the former in Jsi4-band, the detection by our selection criteria is presumably attributed to the 
object at higher redshift. As we will describe in the next section, the fraction of interlopers at this magnitude range is 
expected to be around 40 to 50 %. Thus it is not surprising that half of the objects with redshift information seems to 
be interlopers. The remaining two objects lie close to the edges of the HDF-N and photometric redshift measurement 
is not performed on them. 

In the flanking fields of the HDF-N we find 12 LBG candidates. Two galaxies have been previously identified 
spectroscopically as objects at z ~ 5. They are J123649. 2+621539 (ESI), a screndipitously detected object at z = 
5.19 (Dawson et al. 2002), and B01-174, a broad-line AGN at z = 5.186 discovered in the Chandra Deep Field North 
region by Barger et al. (2002). The images of ESI are shown in figure 3. 

Vice versa, there are 7 galaxies in the HDF-N and its flanking fields, which have been spectroscopically identified as 
4.4 < z < 5.5 but are not included in our candidate list. 3 One object is located close to the edge of a CCD chip and the 
source detection was not executed there. Five galaxies are fainter than our magnitude limit or spatially unresolved 
with a nearby object in our image. The remaining one object is HDF 4—439.0, of which redshift is given as 4.54 
by Stern and Spinrad (1999). The V — I c and I c — z' colors of the object are close to the boundary of our selection 
criteria, and we would fail to select the object probably due to the photometric error. 

The detection rate of our survey for the objects with Jab (814) > 25.0 with spectroscopic redshift between 4.4 < z < 5.5 
in the HDF-N and its flanking field is ~ 1/3. It is consistent with the incompleteness estimated by the Monte Carlo 
simulations described in the next section. 

3.3. Morphology and Size Distribution 

In regard to the effect of the merging event on the star formation activity of the LBGs, it is interesting that the 
HDF-N igi4 image of HDF 4—625.0 at z = 4.58 shows the sign of merging or interaction (see the bottom panels in 
figure 4). If we can figure out the rate of mergers in our sample, we might be able to assess the effect of merging events 
on the star formation of the LBGs. Although our ground-based images have poor angular resolution and is not as deep 
as HDF-N HST/WFPC2 images, we try to examine the fraction of galaxies showing a sign of interaction with other 
galaxies among our LBG candidates. We count the number of objects which have multiple cores in I c and z'-bands 
and are not seen or significantly faint in V^-band among our LBG candidates with I c magnitude between 24 and 25 
(An example is the object with I c = 24.60 in figure 3). Roughly 10 % of the LBG candidates show such feature. This 
fraction should, however, be regarded as a lower limit, since we cannot distinguish close encounters and mergers from 
a single object. The seeing condition of our observation (~ 1", corresponding to ~ 4.4 ft, -1 kpc for z = 5.0, J7m = 0.3 
and = 0.7) is not suitable for the detailed analysis of the sizes and morphologies of the LBG candidates; about 40 

3 There are three objects with a spectroscopic redshift larger than 5.5 in the HDF-N and its flanking fields. All of them are faint and 
not detected in our image. As it is described in the next section, incompleteness of our data gets larger for objects at z 5.5. 
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Fig. 3. Images of the representative LBG candidates, in an order of I c magnitude. From left to right, V, I c and z'-band images 
are shown in each panel. The field of view is 10". 
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Fig. 4. Images of four LBG candidates in the HDF-N region, of which redshifts (either spectroscopic or photometric) are available. 
The field of view is 10". Note that position angles of HDF/WFPC2 images and Suprimc-Cam images arc different, as shown in the 
bottom of the figure. 
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Fig. 5. Apparent I c magnitude against I c — z' color for LBG candidates at z ~ 5. Mean values of 7 C — 2' colors in 0.5 I c mag step are 
shown with filled squares. Each error bar represents the photometric error of the I c — z' color in each 0.5 magnitude bin, measured 
by a simulation using artificial objects (see section 2.4). 

% of the candidates have half light radius comparable to the seeing size. 

3.4- Magnitude and Color Distribution 

In figure 5, apparent I c magnitudes against the color (I c — z') arc plotted. A tendency that bright blue candidates 
are deficient is seen. Other any clear trends are not seen, though the photometric error is larger for faint objects 
(~0.45 mag in 25 < I c < 25.5). Figure 6 shows a distribution of I — z' colors of the LBG candidates with I c < 25.0 
in our search. The I c — z' color of 0.07 mag corresponds to the bluest model LBG spectrum we adopted in section 
2.1, which is regarded here as the extinction free template LBG spectrum. If the redder I c — z' colors originate in 
the internal extinction, we are able to derive a rough estimate of the extinction of each galaxy by comparing with the 
model spectra with different degree of extinction. In the upper abscissa of the figure, we indicate the I c — z' color of 
the model spectra of LBGs at z = 5 with E(B — V) = 0.0 to 0.4 assuming the extinction law by Calzctti et al. (2000). 
As it is seen in figure 2a, the relationship between I c — z' color and dust extinction depends on redshift of an object. 
If we adopt redshift 4.8 instead of 5.0, E(B — V) = 0.0 corresponds to I c — z' ~ 0. The distribution of E(B — V) of 
the LBGs at z ~ 3 estimated by Steidel et al. (1999) is also shown in figure 6 (dashed curve). The distribution of 
E(B — V) based on the template fitting for rest-frame UV to optical SEDs of LBGs at z ~ 3 (Papovich et al. 2001) 
is broadly consistent with that of Steidel et al. (1999). Although the photometric errors for fainter objects in our 
candidates are large and a possible contamination by the emission lines might disturb this distribution, the degree of 
the dust extinction for LBG candidates detected by our search seems to be similar to those at z ~ 3. 

4. The Luminosity Function of the LBG candidates at z ~ 5 

In this section, we try to derive a luminosity function of LBGs at z ~ 5 statistically using our LBG candidates. The 
redshifts of the candidates are not yet confirmed by spectroscopy except for a few objects. However, considering that 
the method works so efficiently at z ~ 3 and z ~ 4 and the color selection criteria adopted here is well examined thanks 
to a large redshift database based on deep spectroscopic and multi-waveband photometric observations in HDF-N and 
its flanking fields, we believe the catalog is reliable enough to discuss statistical nature of them. We would like to make 
follow-up spectroscopy in future, but it would take long years. Under these circumstances, constructing luminosity 
function and discussing its implication to cosmic star formation history statistically is opportune at this stage. In 
deriving the luminosity function of the LBGs at z ~ 5, we should consider the two important issues; contamination 
by interlopers and incompleteness of the sample. In the following procedure of this section we exclude the data in the 
CCD chip at the south-west corner with many bad columns; only five candidates are detected in the chip. 

4-.1. Correction for Contamination by Interlopers 

In section 3 it has been described that our selection criteria are defined to be least contaminated by interlopers via 
redshift database of galaxies in the HDF-N and its flanking fields. While the galaxies in the HDF-N and its flanking 
fields are quite useful for the determination of the selection criteria, they are not suitable for the estimation of a 
number of interlopers contained in our LBG candidates, because the number of these galaxies with available redshift 
is only a few per cent of all the detected sources in our survey. Thus the estimation of the contamination rate based 
on the whole sample of detected objects in our catalog is necessary for the proper correction for interlopers. 
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Fig. 6. Ic — 2' color distribution of LBG candidates at z ~ 5. The objects with I c — z' colors less than —0.5 are included in the 
bin for the bluest color. The ticks at the upper horizontal axis indicate the expected I c — z' colors of a model galaxy at z = 5 with 
E(B — V) from 0.0 to 0.4. See text for the details. The dashed line shows a distribution of E(B — V) for LBGs at z ~ 3 by Steidel 
et al. (1999). 

There are some possible causes of contamination by interlopers other than the photometric errors: the first one is 
interlopers with colors which arc intrinsically similar to LBGs at z > 4.5, and the second one is systematic errors due 
to the indirect z'-band calibration (section 2.3). The fact that there is no object at intermediate redshift for galaxies 
in the CFGRS and only two among 317 galaxies in the photometric redshift catalog of HDF-N by Fernandez-Soto et 
al. (1999) (see section 3.1 and figures 2b and 2c) would certify the validity of our selection criteria (equation (1) and 
(2) in section 3.1). This estimation of contamination by interlopers described below is based on this assumption. In 
regards to the systematic error due to z'-band calibration, it does not affect to our results, because we have defined 
our color selection criteria based on the colors of galaxies at the intermediate redshift in the HDF-N and its flanking 
fields. If I c — z' color distribution is shifted due to the systematic zero point shift in z'-band, the color selection 
criteria should also be shifted in the same direction. So we only consider photometric errors in quantifying the degree 
of contamination below. 

The fraction of interlopers varies with their apparent magnitude, because the larger photometric errors in the 
measurement of fainter objects should increase the probability of their invasion into the z ~ 5 selection window even if 
their intrinsic color is the same as the brighter ones. Thus we should estimate and correct the fraction of the interlopers 
at each magnitude bin. In order to accomplish this task, we employed a resampling (bootstrap) algorithm. First, we 
selected objects at the outside of the selection window in the V — I c and I c — z' color diagram from all of the detected 
objects. Then we divided them by their I c magnitude with a 0.5 mag step. For each magnitude bin, we resampled 
objects randomly allowing the possibility of duplication. The number of objects included in a resampled group was the 
same as the number of objects in the magnitude bin. Then we assigned photometric errors to V — I c and I c — z' colors 
according to the observed I c magnitude and color of each object; thus also according to V and z'-band magnitude. 
The error values in each band were based on the estimation by putting many artificial objects, and the obtained error 
values are consistent with those estimated from the background noise with 1"6 diamter (see section 2.4). The typical 
values for objects with colors close to the border of our color selection criteria were 0.07 mag and 0.11 mag for V — I c 
and I c — z' colors, respectively, for objects with 23.0 < I c < 23.5, and 0.43 mag and 0.45 mag for V — I c and I c — z' 
colors, respectively, for objects with 25.0 < I c < 25.5. The error assignment was made randomly under the assumption 
that photometric errors have a Gaussian distribution. We counted the number of objects of which modified colors 
are in the selection window. We ran this procedure 1,000 times for each magnitude bin and we regard the average of 
the numbers of objects that fall in the selection window as the estimated number of interlopers. We also performed 
the same procedure by adding objects which are placed inside of the selection window but are close to the boundary; 
their color differences from the boundary of the selection criteria are within the photometric errors in both V — I c and 
I c — z' . By this test we intend to estimate a possible maximum number of interlopers. The number of objects within 
the window after adding the random error was roughly twice the number resulted in the first test (in which the objects 
at the outside of the selection window are used) over the all magnitude ranges. Since some (not all, but unknown) 
fraction of the objects within the selection window is considered to be interlopers, the first test should underestimate 
the number of interlopers, while the second test should overestimate it. We took the average of the results of these 
two tests as the expected number of interlopers. The variation of the number densities caused by this uncertainty in 
the estimation of the interlopers is ~ 10 - 30 % for all magnitude bins. We show the number of the interlopers thus 



Lyman Break Galaxies at z ~ 5 



13 



Table 2. The 

ics at z 



number 



of objects 



detected 



survey 



with 



colors expected for galax- 



5, 


and 


corrections 


for interlopers, 


noise 


misidentification, and incompleteness. 


Maf 


mitudc 


Detected 


Expected Number 


Noise 




ranf 


Se (Ic) 


Number 


of Interlopers 


Rate[*] 


N(m) 


(1) 




(2) 


(3) 


(4) 


((2)-(3))*(l-(4)) 



23.0 
23.5 
24.0 
24.5 
25.0 



23.5 
24.0 
24.5 
25.0 
25.5 



6 
25 
61 
91 
122 



1.3 (22%) 
6.6 (26%) 
18.5 (30.3%) 
33.9 (37.3%) 
58.5 (48.0%) 



0.0 

0.0 

0.0 

0.036 

0.172 



4.7 
18.4 
42.5 
55.0 
52.6 
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obtained in table 2. The estimated fraction of interlopers gradually increases along with the magnitude, from 22 % at 
23.0 < I c < 23.5 to 48 % for 25.0 < I c < 25.5. The result is consistent with the fact that no or only a few interlopers 
are seen for spectroscopic or photometric redshift catalog obtained in the HDF-N and its flanking fields (figure 2b and 
2c), since the area of the HDF-N is smaller than our effective surveyed area by a factor of ~ 100. 

Subtraction of these numbers from the numbers of objects within the color selection window gives the expected 
numbers of objects at z ~ 5 if there is no contamination of noise. As described in section 2.4, the cross-identification 
of the sources with the HDF-N catalog shows that in the magnitude range I c < 24.5 the sources are almost noise-free. 
In the fainter magnitude range, we have to multiply the expected number (difference of the values in columns 2 and 

3 in table 2) by the fraction of the real objects among the total list of detected sources (1 minus the values in the 
fourth column in table 2) , because the fraction of noises for the detected sources outside of the color selection window 
must be the same as that in the window. Accordingly, we obtain N(m), the corrected number of objects at z ~ 5 in a 
magnitude range between m±0.25. 

4-2. Correction for Incompleteness 

The other thing that we have to consider for the proper derivation of the luminosity function is the correction for 
the incompleteness of our survey. We should consider two aspects of incompleteness (Steidel et al. 1999). One is 
the detection rate against the magnitude; the detection rate decreases with increasing magnitude, which is the usual 
detection incompleteness. The other one is the rate against the redshift and color; as seen in figure 2a, at a redshift 
range around 4 - 4.5, the detection rate should strongly depend on the color of an object. In addition, at the higher 
redshift (say z > 5.5), the location of the detected sources in figure 2 goes out of the window when they are fainter 
than the detection limit in 1^-band. 

In order to correct these incompleteness, we ran Monte Carlo simulations. These Monte Carlo simulations enable 
us to calculate the "effective volume" of our survey (Steidel et al. 1999), 

V e s(m) = J dzp(m,z)dV/dz 7 (3) 

where p(m,z) is a probability of finding an object with a magnitude of m at a redshift of z, which can be obtained as 
the average weighted over the distribution of SEDs of LBGs. 

First we created sets of artificial objects with various magnitudes and various colors at various redshifts. The number 
of objects created in each set is 200. The magnitude range was taken from 23 mag to 25.5 mag with a 0.5 magnitude 
bin. Their angular sizes were adjusted so that the obtained size distribution of the model objects resembles that of 
real objects at the same magnitude. Their V — I c and I c — z' colors of LBGs at z = 4.0 — 5.9 were calculated using 
the model spectra reproducing those of LBGs described in section 2.1. The redshift step was 0.1. We took objects 
with five different colors, which correspond to E(B — V) of 0.0, 0.1, 0.2, 0.3, and 0.4 for each redshift and magnitude 
using the model SEDs. Thus we created 5 x 20 x 5 = 500 sets of 200 artificial objects for each of 8 CCDs. Then we 
put these artificial objects into our V, I c and z'-band images with a random spatial distribution, and performed the 
object detection in the same way as we did for real objects. Next we counted the number of detected objects which 
fell in our selection window for z ~ 5 objects. The fraction of the objects which satisfied our color selection criteria 
can be regarded as the probability of detecting an object with the magnitude and the color at the redshift. Weighted 
average of the probabilities are calculated using the E(B — V) distribution of LBGs at z ~ 3 by Steidel et al. (1999) 
within the range of E(B — V) = 0.0 - 0.4 (see dashed curve of Fig. 6). As it is described in section 3.4, E(B — V) 
distribution of our LBG candidates is suggested to be similar to that by Steidel et al. (1999). The resulting probability 
p(m,z) averaged over 8 CCD chips is shown in figure 7. 4 The overall decrease in detection rate along the magnitude 

4 The variation of p(m, z) among 8 CCD chips is not large; the variation of the V e fj obtained by equation (3) for each CCD is a few % 
for the bright (23.0 < I c < 24.0) sources, and ~ 35 % for the faintest sources. 



14 



Iwata et al. 




Fig. 7. The detection rate p(m,z) of the galaxies against the redshift of the objects estimated based on the Monte Carlo simulations. 
Each line indicates the detection rate for an I c magnitude range of 23.0 - 23.5, 23.5 - 24.0, 24.0 - 24.5, 24.5 - 25.0 and 25.0 - 25.5 
(from top to bottom). 
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0.45), calculated from I c magnitude by assuming the 



_1 Mpc" 



is clearly seen. At the lower redshift (z = 4 — 4.5), the detection rate decreases because of the cutoff of the selection 
window; the degree of the decline depends on the color and thus the color distribution of LBGs. At the higher redshift 
[z = 5.2 — 5.6), the detection rate decreases because the sources detected in 7 c -band but not in X^-band go out of the 
selection window at the fainter I c magnitude and at the redder I c — z' color. The sum of the effective volumes of 8 
CCD chips obtained for each magnitude range is summarized in table 3 for the two cosmological models. 

4-3. Luminosity Function at z ~ 5 

The information on the contamination of the interlopers and the incompleteness of our search described in the 
previous two subsections allow us to calculate the luminosity function of the Lyman break galaxies at z ~ 5 as, 



$(m) = N{m)/V e s{m), 



(4) 



where N{m) is the number of objects in a magnitude bin of m corrected for the estimated contamination by interlopers 
and noise. The luminosity function thus derived is that at rest UV 1340 A, which redshifts into the central wavelength 
of J c -band (8060 A) for a redshift of 5. In order to compare it with the luminosity functions at z ~ 3 and z ~ 4 obtained 
by Steidel et al. (1999), we calculate absolute AB magnitudes at 1700 A and derive the luminosity function at the 
wavelength, assuming a flat spectrum (in /„) which corresponds to I c — z' color of 0.1 mag in Vega magnitude system. 
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Fig. 8. The luminosity function of the Lyman break galaxies at z ~ 5 (red). Assumed cosmology is S!m = 0.3 and Ha = 0.7 for figure 
8a and Cm = 1.0 and Ha = 0.0 for figure 8b. The vertical error bars for our z ~ 5 data include errors in corrections for interlopers 
and incompleteness, as well as the statistical errors in the number of the candidates assuming the Poisson distribution. The assumed 
rcdshift is z = 5.0. The horizontal error bars indicate the magnitude step (0.5 mag) for all data points. The solid squares show the 
luminosity function without the correction of contamination, and the solid triangles are for a simple selection function which is 1.0 
for a redshift range 4.4 to 5.2 and otherwise regardless of magnitude. The open triangles and open circles represent the luminosity 
functions of LBGs at (z) = 3.04 and (z) = 4.13, respectively, taken from Steidel et al. (1999). The vertical error bars for them only 
show statistical errors. 

Table 3 shows the number density of the LBG candidates in the 0.5 mag step corrected for the incompleteness and the 
contamination. The luminosity function of the z ~ 5 population is presented as red circles in figure 8. In deriving the 
luminosity function, we assume the redshift of 5.0 for all the LBG candidates to calculate their luminosity distance. If 
we use the weighted mean redshift of 4.8 derived from p(m,z) (see figure 7) instead, the absolute magnitude becomes 
0.10 mag fainter than that at z = 5.0. In order to show how much the corrections for contaminations and incompleteness 
affect the luminosity function, we show two alternative number densities with solid squares and triangles in figure 8. 
Solid squares represent the number density of LBGs at z ~ 5 in the case that the correction of contamination is 
skipped; i.e., all objects match with our color selection criteria are assumed to be genuinely at z ~ 5. Solid triangles 
indicate the number density assuming that the detection rate is 1.0 within a redshift range between 4.4 and 5.2 and 
otherwise for all magnitude range instead of using the detection rates estimated by Monte Carlo simulation shown 
in figure 7. In figure 8 we also show the luminosity functions for z ~ 3 and 4 obtained by Steidel et al. (1999) as open 
circles and open triangles. In deriving these luminosity functions, Steidel et al. (1999) assume (z) — 3.04 and (z) = 4.13 
respectively, according to the results of their spectroscopic observations. 

There is no significant difference between the luminosity functions for z ~ 3, 4 and 5 in the bright part. Although the 
change of adopted cosmology makes a slight difference in the relative position of the data points, the general tendency 
remains in both figures. In the fainter part, there is a hint of the decrease of the number density at z ~ 5. It should 
be worth noting here that in figure 8, while the vertical error bars for our z ~ 5 data include errors in corrections for 
interlopers and incompleteness as well as the statistical errors in the number of the candidates assuming the Poisson 
distribution, the error bars for z ~ 3 and 4 by Steidel et al. (1999) only include the statistical errors. Since the 
incompleteness of the z ~ 3 survey is reported to be comparable to or slightly better than ours (Steidel et al. 1999), 
if the uncertainty in the corrections for incompleteness and contamination of the interlopers are considered, the error 
bars for z ~ 3 data points should be larger than those appeared in figure 8. Thus within the current depth and 
uncertainties of the observed data it is hard to conclude whether the discrepancy between the number density for z ~ 5 
and that for z ~ 3 at fainter part is real or not. 

Steidel et al. (1999) and Adelberger and Steidel (2000) made the Schechter function fitting to the luminosity function 
of LBGs at z ~ 3 and derived a set of parameters m* = 24.54, a = -1.57 and (ft* = 4.4 x 10" 3 h 3 mag -1 Mpc -3 under 
the assumption CIm = 0.3 and = 0.7. We made an error-weighted Schechter function fitting for our data for z ~ 5, 
and derived m* = 24.5, a = —1.50 and (ft* = 1.1 x 10~ 3 h 3 mag" 1 Mpc" 3 . In figure 9 we show these two Schechter 
functions ovcrplottcd on the observed data points. 

Meurer et al. (1999) suggest that the existence of the correlation between the rest-frame UV color and luminosity of 
the LBGs at z ~ 3; bright LBGs tend to be red. Ouchi et al. (2002) report that the number density of LBGs at z ~ 4 
shows the excess of red galaxies, which is qualitatively consistent with the result by Meurer et al. (1999). In order 
to see whether such correlation between the UV color and the luminosity exists in our sample, we divide it into two 
subsamples by their I c — z' colors (rest-frame 1340A — 1520A) at the median value of 0.21 mag, and show luminosity 



16 



Iwata et al. 




Fig. 9. Same as figure 8a but only for LBGs at z ~ 3 and 5. Schcchtcr functions fitted to the data are also shown. The dashed line 
indicates a Schechter function with 20% number density of LBGs at z ~ 3. Assumed cosmology is flu = 0.3 and = 0.7. 
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Fig. 10. Luminosity functions of LBGs at z ~ 5 with red UV colors (7 C — z' > 0.21, red points) and blue colors (I c — z' < 0.21, blue 
points). 

functions of them in figure 10. In the bright part, the number density of the red galaxies (I c — z' > 0.21) is slightly 
larger than that of the blue galaxies. This trend can also be seen in figure 5. However, the correlation between the 
luminosity and the color is not as clear as in Ouchi et al. (2002), who performed the same procedure for their LBG 
candidates at z ~ 4 and found that red objects are more than two times abundant in the brighter part of the luminosity 
function. For objects with I c < 24.5 in our sample, the number of red galaxies are ~ 1.14 times larger than that of 
blue galaxies. Although these results do not conflict with the results by Meurer et al. (1999) and Ouchi et al. (2002), 
the correlation of the UV color and luminosity in our z ~ 5 sample seems to be weaker than those at z ~ 3 and z ~ 4. 

5. Discussion 

5.1. Star Formation History of an Individual LBG 

Our results show that the luminosity function of LBGs at z ~ 5 is similar to those at z ~ 3 and at z ~ 4, though there 
may be a slight evolution in the faint part. Does this imply that the LBGs at z ~ 5 are the same LBGs at z ~ 3? Arc 
they continuously forming stars during the period from z~5toz^3 (~ 1 Gyr in our adopted A cosmology)? Shaplcy 
et al. (2001) derived the ages of LBGs at z ~ 3 by the model SED fitting method assuming constant star formation. 
They found that the median age is 320 Myr and that many have ages less than 200 Myr and only 20% of them have 
ages larger than 1 Gyr. The fraction of the 20% depends slightly on the luminosity, but there is no clear tendency 
such as brighter objects have larger ages or vice versa. Therefore most of the LBGs at z ~ 3 were not yet forming stars 
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at z ~ 5 and are not direct descendants of LBGs at z ~ 5, if we suppose the continuous star formation history for an 
individual object. In figure 9 we illustrate this with a Schcchtcr function for 20% number density of LBGs at z ~ 3 as 
a dashed line, by simply assuming that each object has the continuous and constant star formation history with an 
age derived by Shaplcy et al. (2001). If we compare the number densities in the bright part (say Mtjv — 51og/i < —21), 
the number density for LBGs at z ~ 5 is close to that at z ~ 3 and the difference is distinct from the 20% line. The 
same situation is depicted by Ferguson et al. (2002). They also argue that an amount of star formation density at 
z ~ 5 contributed by the LBGs seen at z ~ 3 must be about ten times smaller than that at z ~ 3 for the continuous 
star formation history. 

If the LBGs at z ~ 3 are not the direct descendants of LBGs at z ~ 5, did the LBGs at z ~ 5 stop star formation by 
z ~ 3? Do they fade out at z ~ 3? Papovich et al. (2001) point out the absence of massive, non-star-forming galaxies in 
the HDF-N at z ~ 3. It may indicate that the descendants of LBGs at higher redshift which have stopped star formation 
and become fainter in UV by z ~ 3 are rare. Alternatively, the assumption of the constant star formation may not be 
appropriate; star formation may be sporadic. In this case, the ages derived by the SED fitting method would be taken 
to be those of the most recent starburst. Papovich et al. (2001) test SED fitting using two-component models comprised 
of old and young stellar populations, and claim that two-component fitting generally gives smaller ages for the young 
population than the single-age models. The sporadic star formation and the short duty-cycle of the starburst would 
naturally explain the small ages of young stellar population and the existence of older stellar component as well as the 
similarity of the UV luminosity functions at 3 < z < 5. Supposing the sporadic star formation history for individual 
galaxy seems to be reasonable, because the hierarchical clustering model generally implies merging process must occur 
for LBGs at high redshift. In this case, a burst of star formation may occur during a merging or a galaxy interaction. 
In fact, Nagamine (2002) showed such sporadic star formation history for each galaxy takes place associated with a 
merging in a large-scale hydrodynamic simulation. He also presents a resultant rest-frame U-band luminosity function 
at z = 5. We compare it with our result in the UV, assuming a UV — V color of the model template spectrum with 
a moderate reddening. Although the model luminosity function gives a good agreement with the number density of 
LBGs at the bright part, it shows a significant excess (~1 dex) in the fainter part. However, a color could depend 
on luminosity and this comparison seems to be premature; we need to compare our result with a model luminosity 
function directly derived at UV. 

5.2. Evolution of the UV Luminosity Density 

We can calculate the UV luminosity density at z ~ 5 by integrating the obtained luminosity function. In this 
calculation the integration is made in the magnitude ranges of 23.0 < I c < 25.0 and 23.0 <I C < 25.5. The I c magnitudes 
of 23.0 and 25.5 approximately correspond to Muv — 5\ogh of —22.5 and —20.0 in A-cosmology and —21.7 and —19.2 
in Einstein-de Sitter cosmology, respectively. The former integration range is chosen to match with the limiting 
magnitude of the z ~ 4 sample by Steidel et al. (1999) in terms of absolute magnitude. Results are listed in table 4, 
together with those of z ~ 3 and z ~ 4 by Steidel et al. (1999). As it is expected from the shape of the luminosity 
function in particular at faint part, the resultant UV luminosity density at z ~ 5 is 0.56 - 0.69 times of that at z ~ 3 
in the same absolute magnitude range, depending on the choice of cosmology and the integration range. The UV 
luminosity density seems to be getting smaller at higher redshift, though the difference between z ^ 5 and z ~ 3- 4 is 
comparable to the uncertainty and we cannot rule out the possibility that the UV luminosities are almost constant at 
this redshift range. Madau et al. (1996, 1998) derived UV luminosity densities at 2 < z < 4 using LBGs in the HDF-N. 
Their values are smaller than those by Steidel et al. (1999) by a factor of 2-4. The search field for LBGs by Madau 
et al. (1996, 1998) is much smaller than that by Steidel et al. (1999), and the discrepancy may originate in the small 
search volume in the HDF-N due to cosmic variance (Steidel et al. 1999). Casertano et al. (2000) also found 1.3 - 1.9 
times higher UV luminosity density at 2 < z < 4 in the HST/WFPC2 image of HDF-S as compared with that in the 
HDF-N. 

To examine a degree of uncertainty in deriving the UV luminosity density, we made a calculation of the luminosity 
function and the UV luminosity density using the smaller numbers of interlopers (the first test described in section 
5.1). The resulting UV luminosity density is ~ 1.2 times larger than those listed in table 4 (an increase of 0.08 in 
log scale). In addition, there is also an uncertainty in the estimate of the effective volume. From the chip-to-chip 
variation, the uncertainty of the effective volume for the whole survey area is estimated to be ~ 10%. Taking these two 
uncertainties into account, the upper most possible UV luminosity density is estimated to be about 1.6 times (~ 0.2 
dex) larger than that shown in table 4. The value is comparable to the luminosity densities at z ~ 3 listed in table 4. 

In figure 11 we show the star formation density as a function of redshift. The assumed cosmology is £Im = 0.3 
and Oa = 0-7- The data point at z ~ 5 from our data and the points at z ~ 3 and 4 from Steidel et al. (1999) are 
derived by the integration of the observed number densities for objects down to Mtjv — 5\ogh < —20, as listed in table 
4. For these data points at z > 3, the UV luminosity density at 1500 A is converted to star formation density using 
a conversion factor by Madau et al. (1998), i.e., SFR (M Q yr" 1 ) = L uv (erg s" 1 Hz" 1 ) xl.25 x 10~ 28 . For z ~ 5 
data point this gives SFR of 7.18 x 10" 3 h M Q yr" 1 Mpc -3 (-2.14 in log scale) and 1.30 x 10 -2 h M Q yr" 1 Mpc -3 
(—1.88 in log scale) for A cosmology and Einstein-de Sitter cosmology, respectively. The data points for z ~ 0.5 - 2.0 
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Fig. 11. The star formation density as a function of redshift. Our data point for z ~ 5 LBGs are marked as a red filled circle. Our 
data and those by Steidel et al. (1999) at z ~ 3 and 4 are derived by the integration of the luminosity function with absolute UV 
magnitude M\jy— 5 log h < —20.0 (see table 4). Three data points (shown in filled triangles) for z ~ 0.5 to 2.0 is from Connolly et 
al. (1997), which are adjusted to Cm = 0.3, Qa = 0.7 cosmology and to represent UV luminosities from objects with Afyv — 5 log h 
< —20. The conversion factors from UV luminosity (at rest-frame 1500Afor z ~ 3 to 5 and 2800Afor z < 2) to star formation density 
is from Madau et al. (1998). Filled symbols indicate values without correction for dust extinction. Dust extinction is corrected 
following the prescription by Calzetti et al. (2000) with E(B — V) = 0.15 for all data points, and the values are shown with open 
symbols. Assumed cosmology is Om = 0.3, Qj\ = 0.7. 



are taken from Connolly et al. (1997) which is based on the UV luminosity densities at rest-frame 2800 A. For this 
wavelength we adopt a conversion factor SFR (M© yr _1 ) = Luv (erg s _1 Hz" 1 ) xl.27 x 10 -28 also given by Madau et 
al. (1998). Originally their UV luminosity densities are given by integrating the Schechter UV luminosity function over 
the all luminosity range and calculation is made for Einstein-de Sitter universe. We adopt here a Schechter function 
with a = — 1.3, and for the consistency with data at z > 3, the integration of the UV luminosity function is made 
down to Muv — 51ogft = — 20. The correction for different cosmological parameters is made following the prescription 
by Somerville et al. (2001). In figure 11 we also indicate the star formation density with a simple dust extinction 
correction as open symbols. In correcting dust extinction we follow Steidel ct al. (1999), who use the extinction curve 
by Calzetti et al. (2000) and assume E(B — V) = 0.15 for all data points. As Steidel et al. (1999) remark, the validity 
of such uniform correction is not clear, since the degree of extinction may be redshift dependent, and dust properties 
may change form time to time. As it is described in section 3.4, the E(B — V) distribution estimated from I c — z' 
colors at z ~ 5 is similar to that of galaxies at z ~ 3 (figure 6), although the photometric error is large for faint objects. 
Thus the assumption that the degree of dust extinction is similar from z ~ 3 to 5 might not be inconsistent with our 
observed results for LBGs at z ~ 5, although there may be heavier dust extinction at z <~ 1 (Trcsse et al. 2002). 

We should note that the values of star formation density appeared in figure 11 arc smaller than those in figure 9 of 
Steidel et al. (1999). This is because our integration limit of the luminosity function is about 0.5 mag brighter than 
that of Steidel et al. (1999). The position of data points for z ~ 1.75 is about 0.4 dex higher relative to z ~ 3 in figure 
11, while in Steidel et al. (1999) the difference is ~ 0.2 dex. This is caused by the difference of the adopted slope of 
the fainter part of the luminosity function between z < 2 and z = 3 ~ 4. 

The overall shape of the star formation history presented in figure 11 shows a steep rise from z — to 2, and 
appears to be almost constant or slowly declining at larger redshift. This tendency has been reproduced by recent 
hydrodynamical simulations (e.g., Nagamine et al. 2001; Ascasibar et al. 2002) and semi-analytic models of galaxy 
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formation which take collisional starburst into account (Somerville et al. 2001; Balland et al. 2002). It is hard to make 
a direct comparison of the observed data and these predictions, because the observations (especially for high-redshift 
objects) are restricted to trace only the bright part of the luminosity function. Although it would be desirable if the 
model predictions with an integration range comparable to the observational limit are available, we do not know such 
predictions among the recently published semi-analytic models and numerical simulations; they usually give the total 
star formation density. The shape of the faint end of the luminosity function has a crucial effect on the UV luminosity 
density and may differ according to the redshift. It is also needed to consider the uncertainty of dust extinction in 
the UV luminosity, the correction factor often amounts to more than 3. We might be able to say, however, that if we 
assume that the shape of the UV luminosity function and the degree of dust extinction do not change significantly 
with respect to the redshift, recent semi-analytic models with collisional starburst such as by Somerville et al. (2001) 
and hydrodynamical simulations such as by Ascasibar et al. (2002) are broadly consistent with the observed data 
including our result for z ~ 5. 

5.3. Contribution of Stellar Sources to the Ionizing Photon Density in the Intergalactic Medium 

Finally we briefly comment on the contribution of stellar sources to the hydrogen-ionizing radiation field at z ~ 5. 
Recently Steidel et al. (2001) claimed that the escape fraction f esc , which represents the rate of Lyman continuum 
photons emitted outside from a galaxy, of the LBGs at z ~ 3 is significantly higher than the observational upper limit 
for star-forming galaxies in the local universe; it is comparable to or larger than 0.5 (but Giallongo et al. (2002) made 
a deep spectroscopic observation for two LBGs and placed upper limit of / csc < 0.16). Steidel et al. (2001) estimated 
the contribution of LBGs at z ~ 3 to the ionizing radiation field at 1 ryd to be (1.2 ± 0.3) x 10 26 h erg s _1 Hz -1 
Mpc -3 , which is about 5 times larger than the contribution from quasars at the same redshift (Madau et al. 1999). 
If the rapid decrease of quasar number density at z > 3 (Fan et al. 2001 and references therein) is real, the number 
of ionizing photons radiated from quasars should strongly decline at higher redshift. Madau et al. (1999) estimated 
it drops by a factor of ~2 from z = 3 to z = 5. Provided that the escape fraction of LBGs at z ~ 5 is as large as 0.5, 
our estimate of the UV luminosity density at z ~ 5 implies the ionizing photons originated from star forming galaxies 
are ~5 to 7 times larger than those from quasars at the redshift. This further suggests that the source of the UV 
photons which reionized the Universe at sometime of z > 5 is dominated by stellar origin. It is also interesting that, 
the combination of the estimated numbers of the ionizing photons from LBGs and quasars at z ~ 5 is comparable to 
the number of photons needed to fully ionize a universe considered in Madau et al. (1999) (Einstein-de Sitter universe 
with a clumping factor of 30 and the baryon fraction f2{,h 2 =0.02). However, this coincidence is based on the quite 
uncertain factors such as the quasar number density, the escape fraction of ionizing photons, and the dumpiness of 
the hydrogen gas in the early universe. A deep observation of the rest-frame far-UV wavelengths of LBGs is one of 
the indispensable subjects needed to clarify the nature of the ionizing radiation field at z > 3. 

6. Summary and Conclusion 

We have presented the results of a search for Lyman break galaxies at the redshift range between 4.4 and 5.3 in the 
field including the HDF-North with Suprime-Cam attached to Subaru telescope. Databases for galaxies in the HDF-N 
and its flanking fields enable us to find out the color selection criteria with least foreground contamination. Within 
the surveyed area of 618.6 arcmin 2 we have found 96 LBG candidates with 23.0 < I c < 24.5 mag, and 310 candidates 
with I c < 25.5 mag. There is a hint of the deficiency of bright blue galaxies, although it is not as clear as suggested 
for LBGs at z ~ 3 to 4. 

The contamination of objects at intermediate rcdshifts is estimated employing the resampling algorithm. The 
fraction of contamination amounts to ~20% at bright part of the sample and ~50% at the faintest magnitude range. We 
have also performed Monte Carlo simulations for the evaluation of the incompleteness of the survey. The completeness 
is estimated as a function of the magnitude and the redshift of an object; in the redshift range of 4.5 < z < 5.0, it is 
~80% in the bright part of the sample and ^20% in the faintest magnitude range. Using these corrections, we derive 
the luminosity function of LBGs at z ~ 5 statistically in the absolute UV magnitude range from —23.0 to —20.5 (with 
fiji = 0.3, S1a = 0.7, ho = 0.65). No significant difference is seen between the luminosity function of LBGs at z ~ 5 
and those at z ~ 3 and 4 obtained by Steidel et al. (1999), though there might be a decrease of the number density at 
the fainter magnitude. We have compared the UV luminosity density at the redshift 5 to those at z ~ 3 and 4. If we 
adopt the direct sum of the luminosity function within our observational limit, the star formation density decreases to 
56-69% of that at z ~ 3 estimated by Steidel et al. (1999). This decrease in UV luminosity density at z ~ 5 compared 
to z ~ 3 is due to the smaller number density of faint galaxies at z ^ 5. We cannot rule out the possibility that the 
UV luminosity density is almost constant at z > 3, if we take account for the larger uncertainty in the faint part of 
the luminosity function. 

We have discussed on a plausible evolutionary scenario of LBGs based on our results. The similarity of the luminosity 
functions at redshifts 5 to 3 implies that most of the LBGs at z ~ 5 should have faded out at z ~ 3 and the LBGs at 
z ~ 5 are different galaxies from those seen at z ~ 3, if we take face values for ages of the LBGs at z ~ 3 obtained by the 
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SED fitting in which a continuous star formation in an individual galaxy is assumed. However, if the star formation 
in LBGs is sporadic, the similarity of the luminosity function at z ~ 3 and 5 would be naturally explained. Such 
sporadic star formation has been suggested by hydrodynamical simulations and semi-analytic models with collisional 
starbursts, and the trend of the cosmic star formation history predicted by these studies resembles to that estimated 
from an integration of the UV luminosity density within the observational limit. 

We thank all staff of Subaru telescope during our observations, especially Yutaka Komiyama, for their devoted 
support and technical assistance. We also appreciate Suprime-Cam group members, especially Masami Ouchi, for 
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